Methods and systems for processing media content

ABSTRACT

Methods and systems for processing media content are described and in at least some embodiments utilize a composite or composition, which can be considered as a representation of an editing project as a single track. Editing projects can have one or more tracks, and each track can be associated with one or more data stream sources that can have effects or transitions applied on them. Compositions can be nested inside one another and provide an economical way for complex editing manipulations to take place.

RELATED APPLICATION

This application is a continuation of and claims priority to both U.S.patent application Ser. No. 10/822,032, filed on Apr. 8, 2004 which, inturn, is a divisional of and claims priority to U.S. patent applicationSer. No. 09/731,491, filed Dec. 6, 2000, now U.S. Pat. No. 6,768,499 thedisclosures of which are incorporated by reference herein.

TECHNICAL FIELD

This invention generally relates to processing media content and, moreparticularly, to a system and related interfaces facilitating theprocessing of media content.

BACKGROUND

Recent advances in computing power and related technology have fosteredthe development of a new generation of powerful software applications.Gaming applications, communications applications, and multimediaapplications have particularly benefited from increased processing powerand clocking speeds. Indeed, once the province of dedicated, specialtyworkstations, many personal computing systems now have the capacity toreceive, process and render multimedia objects (e.g., audio and videocontent). While the ability to display (receive, process and render)multimedia content has been around for a while, the ability for astandard computing system to support true multimedia editingapplications is relatively new.

In an effort to satisfy this need, Microsoft Corporation introduced aninnovative development system supporting advanced user-definedmultimedia editing functions. An example of this architecture ispresented in U.S. Pat. No. 5,913,038 issued to Griffiths and commonlyowned by the assignee of the present invention, the disclosure of whichis expressly incorporated herein by reference.

In the '038 patent, Griffiths introduced the an application programinterface which, when exposed to higher-level development applications,enables a user to graphically construct a multimedia processing projectby piecing together a collection of “filters” exposed by the interface.The interface described therein is referred to as a filter graphmanager. The filter graph manager controls the data structure of thefilter graph and the way data moves through the filter graph. The filtergraph manager provides a set of component object model (COM) interfacesfor communication between a filter graph and its application. Filters ofa filter graph architecture are preferably implemented as COM objects,each implementing one or more interfaces, each of which contains apredefined set of functions, called methods. Methods are called by anapplication program or other component objects in order to communicatewith the object exposing the interface. The application program can alsocall methods or interfaces exposed by the filter graph manager object.

Filter graphs work with data representing a variety of media (ornon-media) data types, each type characterized by a data stream that isprocessed by the filter components comprising the filter graph. A filterpositioned closer to the source of the data is referred to as anupstream filter, while those further down the processing chain isreferred to as a downstream filter. For each data stream that the filterhandles it exposes at least one virtual pin (i.e., distinguished from aphysical pin such as one might find on an integrated circuit). A virtualpin can be implemented as a COM object that represents a point ofconnection for a unidirectional data stream on a filter. Input pinsrepresent inputs and accept data into the filter, while output pinsrepresent outputs and provide data to other filters. Each of the filtersinclude at least one memory buffer, wherein communication of the mediastream between filters is often accomplished by a series of “copy”operations from one filter to another.

As introduced in Griffiths, a filter graph has three different types offilters: source filters, transform filters, and rendering filters. Asource filter is used to load data from some source; a transform filterprocesses and passes data; and a rendering filter renders data to ahardware device or other locations (e.g., saved to a file, etc.). Anexample of a filter graph for a simplistic media rendering process ispresented with reference to FIG. 1.

FIG. 1 graphically illustrates an example filter graph for renderingmedia content. As shown, the filter graph 100 is comprised of aplurality of filters 102-114, which read, process (transform) and rendermedia content from a selected source file. As shown, the filter graphincludes each of the types of filters described above, interconnected ina linear fashion.

Products utilizing the filter graph have been well received in themarket as it has opened the door to multimedia editing using otherwisestandard computing systems. It is to be appreciated, however, that theconstruction and implementation of the filter graphs are computationallyintensive and expensive in terms of memory usage. Even the most simpleof filter graphs requires and abundance of memory to facilitate the copyoperations required to move data between filters. Complex filter graphscan become unwieldy, due in part to the linear nature of prior artfilter graph architecture. Moreover, it is to be appreciated that thefilter graphs themselves consume memory resources, thereby compoundingthe issue introduced above.

Thus, what is required is a filter graph architecture which reduces thecomputational and memory resources required to support even the mostcomplex of multimedia projects. Indeed, what is required is adynamically reconfigurable multimedia editing system and relatedmethods, unencumbered by the limitations described above. Just such asystem and methods are disclosed below.

SUMMARY

Methods and systems for processing media content are described and in atleast some embodiments utilize a composite or composition, which can beconsidered as a representation of an editing project as a single track.Editing projects can have one or more tracks, and each track can beassociated with one or more data stream sources that can have effects ortransitions applied on them. Compositions can be nested inside oneanother and provide an economical way for complex editing manipulationsto take place.

BRIEF DESCRIPTION OF THE DRAWINGS

The same reference numbers are used throughout the figures to referencelike components and features.

FIG. 1 is a graphical representation of a conventional filter graphrepresenting a user-defined development project.

FIG. 2 is a block diagram of a computing system incorporating theteachings of the described embodiment.

FIG. 3 is a block diagram of an example software architectureincorporating the teachings of the described embodiment.

FIG. 4 is a graphical illustration of an example software-enabled matrixswitch, according to an exemplary embodiment.

FIG. 5 is a graphical representation of a data structure comprising aprogramming grid to selectively couple one or more of a scalableplurality of input pins to a scalable plurality of output pins of thematrix switch filter, in accordance with one aspect of the describedembodiment.

FIG. 6 is a graphical illustration denoting shared buffer memory betweenfilters, according to one aspect of the described embodiment.

FIG. 7 is a flow chart of an example method for generating a filtergraph, in accordance with one aspect of the described embodiment.

FIG. 8 is a flow chart of an example method for negotiating bufferrequirements between at least two adjacent filters, according to oneaspect of the described embodiment.

FIG. 9 graphically illustrates an overview of a process that takes auser-defined editing project and composites a data structure that can beused to program the matrix switch.

FIG. 10 graphically illustrates the project of FIG. 9 in greater detail.

FIG. 11 shows an exemplary matrix switch dynamically generated insupport of the project developed in FIGS. 9 and 10, according to onedescribed embodiment.

FIG. 12 illustrates a graphic representation of an exemplary datastructure that represents the project of FIG. 10, according to onedescribed embodiment.

FIGS. 13-18 graphically illustrate various states of a matrix switchprogramming grid at select points in processing the project of FIGS. 9and 10 through the matrix switch, in accordance with one describedembodiment.

FIG. 19 is a flow chart of an example method for processing mediacontent, in accordance with one described embodiment.

FIG. 20 illustrates an example project with a transition and an effect,in accordance with one described embodiment.

FIG. 21 shows an exemplary data structure in the form of a hierarchicaltree that represents the project of FIG. 20.

FIGS. 22 and 23 graphically illustrate an example matrix switchprogramming grid associated with the project of FIG. 20 at select pointsin time, according to one described embodiment.

FIG. 24 shows an example matrix switch dynamically generated andconfigured as the grid of FIGS. 22 and 23 was being processed, inaccordance with one described embodiment.

FIG. 25 shows an exemplary project in accordance with one describedembodiment.

FIG. 26 graphically illustrates an example audio editing project,according to one described embodiment.

FIG. 27 depicts an example matrix switch programming grid associatedwith the project of FIG. 26.

FIG. 28 shows an example matrix switch dynamically generated andconfigured in accordance with the programming grid of FIG. 27 to performthe project of FIG. 26, according to one described embodiment.

FIG. 29 illustrates an exemplary media processing project incorporatinganother media processing project as a composite, according to yetanother described embodiment.

FIG. 30 graphically illustrates an example data structure in the form ofa hierarchical tree structure that represents the project of FIG. 29.

FIGS. 31-37 graphically illustrate various matrix switch programminggrid states at select points in generating and configuring the matrixswitch to implement the media processing of FIG. 29.

FIG. 38 illustrates an example matrix switch suitable for use in themedia processing project of FIG. 29, according to one describedembodiment.

FIG. 38 a graphically illustrates an example data structure in the formof a hierarchical tree structure that represents a project that isuseful in understanding composites in accordance with the describedembodiments.

FIG. 39 is a flow diagram that describes steps in a method in accordancewith one described embodiment.

DETAILED DESCRIPTION Related Applications

This application is related to the following commonly-filed U.S. PatentApplications, all of which are commonly assigned to Microsoft Corp., thedisclosures of which are incorporated by reference herein:

Application Ser. No. 09/731,560, entitled “An Interface and RelatedMethods for Reducing Source Accesses in a Development System”, namingDaniel J. Miller and Eric H. Rudolph as inventors;

Application Ser. No. 09/732,084, entitled “A System and RelatedInterfaces Supporting the Processing of Media Content”, naming Daniel J.Miller and Eric H. Rudolph as inventors;

Application Ser. No. 09/731,490, entitled “A System and Related Methodsfor Reducing Source Filter Invocation in a Development Project”, namingDaniel J. Miller and Eric H. Rudolph as inventors;

Application Ser. No. 09/732,452, entitled “A System and Related Methodsfor Reducing Memory Requirements of a Media Processing System”, namingDaniel J. Miller and Eric H. Rudolph as inventors;

Application Ser. No. 09/731,529, entitled “A System and Related Methodsfor Reducing the Instances of Source Files in a Filter Graph”, namingDaniel J. Miller and Eric H. Rudolph as inventors;

Application Ser. No. 09/732,087, entitled “An Interface and RelatedMethods for Dynamically Generating a Filter Graph in a DevelopmentSystem”, naming Daniel J. Miller and Eric H. Rudolph as inventors;

Application Ser. No. 09/732,090, entitled “A System and Related Methodsfor Processing Audio Content in a Filter Graph”, naming Daniel J. Millerand Eric H. Rudolph as inventors;

Application Ser. No. 09/732,085, entitled “A System and Methods forGenerating an Managing Filter Strings in a Filter Graph”, naming DanielJ. Miller and Eric H. Rudolph as inventors;

Application Ser. No. 09/731,563, entitled “Systems for Managing MultipleInputs and Methods and Systems for Processing Media Content”, namingDaniel J. Miller and Eric H. Rudolph as inventors;

Application Ser. No. 09/731,892, entitled “Methods and Systems forImplementing Dynamic Properties on Objects that Support Only StaticProperties”, naming Daniel J. Miller and David Maymudes as inventors;

Application Ser. No. 09/732,089, entitled “Methods and Systems forEfficiently Processing Compressed and Uncompressed Media Content”,naming Daniel J. Miller and Eric H. Rudolph as inventors;

Application Ser. No. 09/731,581, entitled “Methods and Systems forEffecting Video Transitions Represented By Bitmaps”, naming Daniel J.Miller and David Maymudes as inventors;

Application Ser. No. 09/732,372, entitled “Methods and Systems forMixing Digital Audio Signals”, naming Eric H. Rudolph as inventor;

Application Ser. No. 09/732,086, entitled “Methods and Systems forProcessing Multi-media Editing Projects”, naming Eric H. Rudolph asinventor.

Various described embodiments concern an application program interfaceassociated with a development system. According to one exampleimplementation, the interface is exposed to a media processingapplication to enable a user to dynamically generate complex mediaprocessing tasks, e.g., editing projects. In the discussion herein,aspects of the invention are developed within the general context ofcomputer-executable instructions, such as program modules, beingexecuted by one or more conventional computers. Generally, programmodules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. Moreover, those skilled in the art will appreciatethat the invention may be practiced with other computer systemconfigurations, including hand-held devices, personal digitalassistants, multiprocessor systems, microprocessor-based or programmableconsumer electronics, network PCs, minicomputers, mainframe computers,and the like. In a distributed computer environment, program modules maybe located in both local and remote memory storage devices. It is noted,however, that modification to the architecture and methods describedherein may well be made without deviating from spirit and scope of thepresent invention. Moreover, although developed within the context of amedia processing system paradigm, those skilled in the art willappreciate, from the discussion to follow, that the application programinterface may well be applied to other development systemimplementations. Thus, the media processing system described below isbut one illustrative implementation of a broader inventive concept.

Example System Architecture

FIG. 2 illustrates an example of a suitable computing environment 200 onwhich the system and related methods for processing media content may beimplemented.

It is to be appreciated that computing environment 200 is only oneexample of a suitable computing environment and is not intended tosuggest any limitation as to the scope of use or functionality of themedia processing system. Neither should the computing environment 200 beinterpreted as having any dependency or requirement relating to any oneor combination of components illustrated in the exemplary computingenvironment 200.

The media processing system is operational with numerous other generalpurpose or special purpose computing system environments orconfigurations. Examples of well known computing systems, environments,and/or configurations that may be suitable for use with the mediaprocessing system include, but are not limited to, personal computers,server computers, thin clients, thick clients, hand-held or laptopdevices, multiprocessor systems, microprocessor-based systems, set topboxes, programmable consumer electronics, network PCs, minicomputers,mainframe computers, distributed computing environments that include anyof the above systems or devices, and the like.

In certain implementations, the system and related methods forprocessing media content may well be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, etc. that performparticular tasks or implement particular abstract data types. The mediaprocessing system may also be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including memory storage devices.

In accordance with the illustrated example embodiment of FIG. 2computing system 200 is shown comprising one or more processors orprocessing units 202, a system memory 204, and a bus 206 that couplesvarious system components including the system memory 204 to theprocessor 202.

Bus 206 is intended to represent one or more of any of several types ofbus structures, including a memory bus or memory controller, aperipheral bus, an accelerated graphics port, and a processor or localbus using any of a variety of bus architectures. By way of example, andnot limitation, such architectures include Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnects (PCI) buss also known asMezzanine bus.

Computer 200 typically includes a variety of computer readable media.Such media may be any available media that is locally and/or remotelyaccessible by computer 200, and it includes both volatile andnon-volatile media, removable and non-removable media.

In FIG. 2, the system memory 204 includes computer readable media in theform of volatile, such as random access memory (RAM) 210, and/ornon-volatile memory, such as read only memory (ROM) 208. A basicinput/output system (BIOS) 212, containing the basic routines that helpto transfer information between elements within computer 200, such asduring start-up, is stored in ROM 208. RAM 210 typically contains dataand/or program modules that are immediately accessible to and/orpresently be operated on by processing unit(s) 202.

Computer 200 may further include other removable/non-removable,volatile/non-volatile computer storage media. By way of example only,FIG. 2 illustrates a hard disk drive 228 for reading from and writing toa non-removable, non-volatile magnetic media (not shown and typicallycalled a “hard drive”), a magnetic disk drive 230 for reading from andwriting to a removable, non-volatile magnetic disk 232 (e.g., a “floppydisk”), and an optical disk drive 234 for reading from or writing to aremovable, non-volatile optical disk 236 such as a CD-ROM, DVD-ROM orother optical media. The hard disk drive 228, magnetic disk drive 230,and optical disk drive 234 are each connected to bus 206 by one or moreinterfaces 226.

The drives and their associated computer-readable media providenonvolatile storage of computer readable instructions, data structures,program modules, and other data for computer 200. Although the exemplaryenvironment described herein employs a hard disk 228, a removablemagnetic disk 232 and a removable optical disk 236, it should beappreciated by those skilled in the art that other types of computerreadable media which can store data that is accessible by a computer,such as magnetic cassettes, flash memory cards, digital video disks,random access memories (RAMs), read only memories (ROM), and the like,may also be used in the exemplary operating environment.

A number of program modules may be stored on the hard disk 228, magneticdisk 232, optical disk 236, ROM 208, or RAM 210, including, by way ofexample, and not limitation, an operating system 214, one or moreapplication programs 216 (e.g., multimedia application program 224),other program modules 218, and program data 220. In accordance with theillustrated example embodiment of FIG. 2, operating system 214 includesan application program interface embodied as a render engine 222. Aswill be developed more fully below, render engine 222 is exposed tohigher-level applications (e.g., 216) to automatically assemble filtergraphs in support of user-defined development projects, e.g., mediaprocessing projects. Unlike conventional media processing systems,however, render engine 222 utilizes a scalable, dynamicallyreconfigurable matrix switch to reduce filter graph complexity, therebyreducing the computational and memory resources required to complete adevelopment project. Various aspects of the innovative media processingsystem represented by a computer 200 implementing the innovative renderengine 222 will be developed further, below.

Continuing with FIG. 2, a user may enter commands and information intocomputer 200 through input devices such as keyboard 238 and pointingdevice 240 (such as a “mouse”). Other input devices may include aaudio/video input device(s) 253, a microphone, joystick, game pad,satellite dish, serial port, scanner, or the like (not shown). These andother input devices are connected to the processing unit(s) 202 throughinput interface(s) 242 that is coupled to bus 206, but may be connectedby other interface and bus structures, such as a parallel port, gameport, or a universal serial bus (USB).

A monitor 256 or other type of display device is also connected to bus206 via an interface, such as a video adapter 244. In addition to themonitor, personal computers typically include other peripheral outputdevices (not shown), such as speakers and printers, which may beconnected through output peripheral interface 246.

Computer 200 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer250. Remote computer 250 may include many or all of the elements andfeatures described herein relative to computer 200 including, forexample, render engine 222 and one or more development applications 216utilizing the resources of render engine 222.

As shown in FIG. 2. computing system 200 is communicatively coupled toremote devices (e.g., remote computer 250) through a local area network(LAN) 251 and a general wide area network (WAN) 252. Such networkingenvironments are commonplace in offices, enterprise-wide computernetworks, intranets, and the Internet.

When used in a LAN networking environment, the computer 200 is connectedto LAN 251 through a suitable network interface or adapter 248. Whenused in a WAN networking environment, the computer 200 typicallyincludes a modem 254 or other means for establishing communications overthe WAN 252. The modem 254, which may be internal or external, may beconnected to the system bus 206 via the user input interface 242, orother appropriate mechanism.

In a networked environment, program modules depicted relative to thepersonal computer 200, or portions thereof, may be stored in a remotememory storage device. By way of example, and not limitation, FIG. 2illustrates remote application programs 216 as residing on a memorydevice of remote computer 250. It will be appreciated that the networkconnections shown and described are exemplary and other means ofestablishing a communications link between the computers may be used.

Turning next to FIG. 3, a block diagram of an example development systemarchitecture is presented, in accordance with one embodiment of thepresent invention. In accordance with the illustrated example embodimentof FIG. 3, development system 300 is shown comprising one or moreapplication program(s) 216 coupled to render engine 222 via anappropriate communications interface 302. As used herein, applicationprogram(s) 216 are intended to represent any of a wide variety ofapplications which may benefit from use of render engine 222 such as,for example a media processing application 224.

The communications interface 302 is intended to represent any of anumber of alternate interfaces used by operating systems to exposeapplication program interface(s) to applications. According to oneexample implementation, interface 302 is a component object model (COM)interface, as used by operating systems offered by MicrosoftCorporation. As introduced above, COM interface 302 provides a means bywhich the features of the render engine 222, to be described more fullybelow, are exposed to an application program 216.

In accordance with the illustrated example implementation of FIG. 3,render engine 222 is presented comprising source filter(s) 304A-N,transform filter(s) 306A-N and render filter 310, coupled togetherutilizing virtual pins to facilitate a user-defined media processingproject. According to one implementation, the filters of system 300 aresimilar to the filters exposed in conventional media processing systems.According to one implementation, however, filters are not coupled viasuch interface pins. Rather, alternate implementations are envisionedwherein individual filters (implemented as objects) make calls to otherobjects, under the control of the render engine 222, for the desiredinput. Unlike conventional systems, however, render engine 222 exposes ascalable, dynamically reconfigurable matrix switch filter 308,automatically generated and dynamically configured by render engine 222to reduce the computational and memory resource requirements oftenassociated with development projects. As introduced above, the pins(input and/or output) are application interface(s) designed tocommunicatively couple other objects (e.g., filters).

In accordance with the example implementation of a media processingsystem, an application communicates with an instance of render engine222 when the application 216 wants to process streaming media content.Render engine 222 selectively invokes and controls an instance of filtergraph manager (not shown) to automatically create a filter graph byinvoking the appropriate filters (e.g., source, transform andrendering). As introduced above, the communication of media contentbetween filters is achieved by either (1) coupling virtual output pinsof one filter to the virtual input pins of requesting filter; or (2) byscheduling object calls between appropriate filters to communicate therequested information. As shown, source filter 304 receives streamingdata from the invoking application or an external source (not shown). Itis to be appreciated that the streaming data can be obtained from a fileon a disk, a network, a satellite feed, an Internet server, a videocassette recorder, or other source of media content. As introducedabove, transform filter(s) 306 take the media content and processes itin some manner, before passing it along to render filter 310. As usedherein, transform filter(s) 306 are intended to represent a wide varietyof processing methods or applications that can be performed on mediacontent. In this regard, transform filter(s) 306 may well include asplitter, a decoder, a sizing filter, a transition filter, an effectsfilter, and the like. The function of each of these filters is describedmore fully in the Griffiths application, introduced above, and generallyincorporated herein by reference. The transition filter, as used herein,is utilized by render engine 222 to transition the rendered output froma first source to a second source. The effect filter is selectivelyinvoked to introduce a particular effect (e.g., fade, wipe, audiodistortion, etc.) to a media stream.

In accordance with one aspect of the embodiment, to be described morefully below, matrix switch filter 308 selectively passes media contentfrom one or more of a scalable plurality of input(s) to a scalableplurality of output(s). Moreover, matrix switch 308 also supportsimplementation of a cascaded architecture utilizing feedback paths,i.e., wherein transform filters 306B, 306C, etc. coupled to the outputof matrix switch 308 are dynamically coupled to one or more of thescalable plurality of matrix switch input(s). An example of thiscascaded filter graph architecture is introduced in FIG. 3, and furtherexplained in example implementations, below.

Typically, media processed through source, transform and matrix switchfilters are ultimately passed to render filter 310, which provides thenecessary interface to a hardware device, or other location that acceptsthe renderer output format, such as a memory or disk file, or arendering device.

FIG. 4 is a graphical illustration of an example software-enabled matrixswitch 308, according to one example embodiment of the presentinvention. As shown, the matrix switch 308 is comprised of a scalableplurality of input(s) 402 and a scalable plurality of output(s) 404,wherein any one or more of the input(s) 402 may be iteratively coupledto any one or more of the output(s) 404, based on the content of thematrix switch programming grid 406, automatically generated by renderengine 222. According to an alternate implementation introduced above,switch matrix 308 is programmed by render engine 222 to dynamicallygenerate object calls to communicate media content between filters. Inaddition, according to one implementation, matrix switch 308 includes aplurality of input/output (I/O) buffers 408, as well as means formaintaining source, or media time 410 and/or timeline, or project time412. It is to be appreciated, however, that in alternate implementationsmatrix switch 308 does not maintain both source and project times,relying on an upstream filter to convert between these times. As will bedeveloped more fully below, matrix switch 308 dynamically couples one ormore of the scalable plurality of inputs 402 to one or more of thescalable plurality of outputs 404 based, at least in part, on the mediatime 410 and/or the project time 412 and the content of matrix switchprogramming grid 406. In this regard, matrix switch 308 may becharacterized as time-aware, supporting such advanced editing featuresas searching/seeking to a particular point (e.g., media time) in themedia content, facilitating an innovative buffering process utilizingI/O buffers 408 to facilitate look-ahead processing of media content,and the like. Thus, it will be appreciated given the discussion tofollow that introduction of the matrix switch 308 provides a user withan editing flexibility that was heretofore unavailable in a personalcomputer-based media processing system.

As introduced above, the inputs 402 and outputs 404 of matrix switch 308are interfaces which facilitate the time-sensitive routing of data(e.g., media content) in accordance with a user-defined developmentproject. Matrix switch 308 has a scalable plurality of inputs 402 andoutputs 404, meaning that the number of inputs 402 and outputs 404 areindividually generated to satisfy a given editing project. Insofar aseach of the inputs/outputs (I/O) has an associated transfer buffer(preferably shared with an adjacent filter) to communicate mediacontent, the scalability of the input/output serves to reduce theoverall buffer memory consumed by an editing project. According to oneimplementation, output 1 is generally reserved as a primary output,e.g., coupled to a rendering filter (not shown).

According to one implementation, for each input 402 and output 404,matrix switch 308 attempts to be the allocator, or manager of the bufferassociated with the I/O(s) shared with adjacent filters. One reason isto ensure that all of the buffers are of the same size and share commonattributes so that a buffer associated with any input 402 may be sharedwith any output 404, thereby reducing the need to copy memory contentsbetween individual buffers associated with such inputs/outputs. Ifmatrix switch 308 cannot be an allocator for a given output (404),communication from an input (402) to that output is performed using aconventional memory copy operation between the individual buffersassociated with the select input/output.

As introduced above, the matrix switch programming grid 406 isdynamically generated by render engine 222 based, at least in part, onthe user-defined development project. As will be developed below, renderengine 222 invokes an instance of filter graph manager to assembles atree structure of an editing project, noting dependencies betweensource, filters and time to dynamically generate the programming grid406. A data structure comprising an example programming grid 406 isintroduced with reference to FIG. 5, below.

Turning briefly to FIG. 5, a graphical representation of a datastructure comprising an example programming grid 406 is presented, inaccordance with one embodiment of the present invention. In accordancewith the illustrated example embodiment of FIG. 5, programming grid 406is depicted as a two-dimensional data structure comprising a columnalong the y-axis 502 of the grid denoting input pins associated with acontent chain (e.g., series of filters to process media content) of thedevelopment project. The top row along the x-axis 504 of the datastructure denotes project time. With these grid “borders”, the body 506of the grid 406 is populated with output pin assignments, denoting whichinput pin is coupled to which output pin during execution of thedevelopment project. In this way, render engine 222 dynamicallygenerates and facilitates matrix switch 308. Those skilled in the artwill appreciate, however, that data structures of greater or lessercomplexity may well be used in support of the programming grid 406without deviating from the spirit and scope of the present invention.

Returning to FIG. 4, matrix switch 308 is also depicted with a pluralityof input/output buffers 408, shared among all of the input(s)/ouptut(s)(402, 404) to facilitate advanced processing features. That is, whilenot required to implement the core features of matrix switch 308, I/Obuffers 408 facilitate a number of innovative performance enhancingfeatures to improve the performance (or at least the user's perceptionof performance) of the processing system, thereby providing an improveduser experience. According to one implementation, I/O buffers 408 areseparate from the buffers assigned to each individual input and outputpin in support of communication through the switch. According to oneimplementation, I/O buffers 408 are primarily used to foster look-aheadprocessing of the project. Assume, for example, that a large portion ofthe media processing project required only 50% of the availableprocessing power, while some smaller portion required 150% of theavailable processing power. Implementation of the shared I/O buffers 408enable filter graph manager to execute tasks ahead of schedule andbuffer this content in the shared I/O buffers 408 until required. Thus,when execution of the filter graph reaches a point where more than 100%of the available processing power is required, the processing system cancontinue to supply content from the I/O buffers 408, while the systemcompletes execution of the CPU-intensive tasks. If enough shared bufferspace is provided, the user should never know that some tasks were notperformed in real-time. According to one implementation, shared buffers408 are dynamically split into two groups by render engine 222, a firstgroup supports the input(s) 402, while a second (often smaller) group isused in support of a primary output (e.g., output pin 1) to facilitate asecond, independent output processing thread. The use of an independentoutput buffers the render engine from processing delays that might occurin upstream and/or downstream filters, as discussed above. It will beappreciated by those skilled in the art that such that matrix switch 308and the foregoing described architecture beneficially suited to supportmedia streaming applications.

As introduced above, the filter graph is time-aware in the sense thatmedia (source) time and project execution time are maintained. Accordingto one implementation, matrix switch 308 maintains at least the projectclock, while an upstream filter maintains the source time, convertingbetween source and project time for all downstream filters (i.e.,including the matrix switch 308). According to one implementation, theframe rate converter filter of a filter graph is responsible forconverting source time to project time, and vice versa, i.e., supportingrandom seeks, etc. Alternatively, matrix switch 308 utilizes anintegrated set of clock(s) to independently maintain project and mediatimes.

Having introduced the architectural and operational elements of matrixswitch filter 308, FIG. 6 graphically illustrates an example filtergraph implementation incorporating the innovative matrix switch 308. Inaccordance with the illustrated example embodiment, filter graph 600 isgenerated by render engine 222 in response to a user defined developmentproject. Unlike the lengthy linear filter graphs typical of conventiondevelopment systems however, filter graph 600 is shown incorporating amatrix switch filter 308 to recursively route the pre-processed content(e.g., through filters 602, 606, 610, 614 and 618, described more fullybelow) through a user-defined number of transform filters including, forexample, transition filter(s) 620 and effects filter(s) 622. Moreover,as will be developed more fully below, the scalable nature of matrixswitch filter 308 facilitates such iterative processing for any numberof content threads, tracks or compositions.

According to one implementation, a matrix switch filter 308 can onlyprocess one type of media content, of the same size and at the sameframe-rate (video) or modulation type/schema (audio). Thus, FIG. 6 isdepicted comprising pre-processing filters with a parser filter 606 toseparate, independent content type(s) (e.g., audio content and videocontent), wherein one of the media types would be processed along adifferent path including a separate instance of matrix switch 308. Thus,in accordance with the illustrated example embodiment of a mediaprocessing system, processing multimedia content including audio andvideo would utilize two (2) matrix switch filters 308, one dedicated toaudio processing (not shown) and one dedicated to video processing. Thatis not to say, however, that multiple switch filters 308 could not beused (e.g., two each for audio and video) for each content type inalternate implementations. Similarly, it is anticipated that inalternate implementations a matrix switch 308 that accepts multiplemedia types could well be used without deviating from the spirit andscope of the present invention.

In addition filter graph 600 includes a decoder filter 610 to decode themedia content. Resize filter 614 is employed when matrix switch 308 isto receive content from multiple sources, ensuring that the size of thereceived content is the same, regardless of the source. According to oneimplementation, resize filter 614 is selectively employed in videoprocessing paths to adjust the media size of content from one or moresources to a user-defined level. Alternatively, resizer filter 614adjusts the media size to the largest size provided by any one or moremedia sources. That is, if, for example, render engine 222 identifiesthe largest required media size (e.g., 1270×1040 video pixels per frame)and, for any content source not providing content at this size, thecontent is modified (e.g., stretched, packed, etc.) to fill this sizerequirement. The frame rate converter (FRC) and pack filter 618,introduced above, ensures that video content from the multiple sourcesis arriving at the same frame rate, e.g., ten (10) frames per second. Asintroduced above, the FRC also maintains the distinction between sourcetime and project time.

In accordance with one aspect of the present invention, filter graph 600is depicted utilizing a single, negotiated buffer 604, 608, 612, 616,etc. between adjacent filters. In this regard, render engine 222 reducesthe buffer memory requirements in support of a development project.

From the point of pre-processing (filters 602, 606, 610, 614, 618),rather than continue a linear filter graph incorporating all of thetransition 620 and effect 622 filter(s), render engine 222 utilizes acascade architecture, recursively passing media content through thematrix switch 308 to apply to the transform filter(s) (e.g., 620, 622,etc.) to complete the execution of the development project. It will beappreciated by those skilled in the art that the ability to recursivelypass media content through one or more effect and/or transition filtersprovided by the matrix switch filter 308 greatly reduces the perceivedcomplexity of otherwise large filter graphs, while reducing memory andcomputational overhead.

Turning to FIG. 7, a flow chart of an example method for generating afilter graph is presented, in accordance with one aspect of the presentinvention. The method 700 begins with block 702 wherein render engine222 receives an indication to generate a filter graph representing auser-defined development project (e.g., a media editing project).According to one example implementation, the indication is received froman application 224 via COM interface(s) 302.

In block 704, render engine 222 facilitates generation of the editingproject, identifying the number and type of media sources selected bythe user. In block 706, based at least in part on the number and/or typeof media sources, filter graph manger 222 exposes source, transform andrendering filter(s) to effect a user defined media processing project,while beginning to establish a programming grid 406 for the matrixswitch filter 308.

In block 708, reflecting user editing instructions, render engine 222completes the programming grid 406 for matrix switch 308, identifyingwhich inputs 402 are to be coupled to which outputs 404 at particularproject times.

Based, at least in part, on the programming grid 406 render engine 222generates a matrix switch filter 308 with an appropriate number of input402 and output 404 pins to effect the project, and assembles the filtergraph, block 710.

In block 712, to reduce the buffer memory requirements for theprocessing project, the render engine 222 instructs the filterspopulating the filter graph to (re)negotiate buffer memory requirementsbetween filters. That is, adjacent filters attempt to negotiate a sizeand attribute standard so that a single buffer can be utilized to coupleeach an output pin of one filter to an input pin of a downstream filter.An example implementation of the buffer negotiation process of block 712is presented in greater detail with reference to FIG. 8.

Turning briefly to FIG. 8, an example method of negotiating bufferrequirements between adjacent filters is presented, in accordance withone example implementation of the present invention. Once the finalconnection is established to matrix switch 308, matrix switch 308identifies the maximum buffer requirements for any filter coupled to anyof its pins (input 402 and/or output 404), block 802. According to oneimplementation, the maximum buffer requirements are defined as thelowest common multiple of buffer alignment requirements, and the maximumof all the pre-fix requirements of the filter buffers.

In block 804, matrix switch 308 selectively removes one or more existingfilter connections to adjacent filters. Matrix switch 308 thenreconnects all of its pins to adjacent filters using a common buffersize between each of the pins, block 806. In block 808, matrix switch308 negotiates to be the allocator for all of its pins (402, 404). Ifthe matrix switch 308 cannot, for whatever reason, be the allocator forany of its input pins 402 minimal loss to performance is encountered, asthe buffer associated with the input pin will still be compatible withany downstream filter (i.e., coupled to an output pin) and, thus, thebuffer can still be passed to the downstream filter without requiring amemory copy operation. If, however, matrix switch 308 cannot be anallocator for one of its output pins 404, media content must then betransferred to at least the downstream filter associated with thatoutput pin using a memory copy operation, block 810.

In block 812, once the matrix switch 308 has re-established itsconnection to adjacent filters, render engine 222 restores theconnection in remaining filters using negotiated buffer requirementsemanating from the matrix switch filter 308 buffer negotiations. Oncethe connections throughout the filter graph have been reconnected, theprocess continues with block 714 of FIG. 7.

In block 714 (FIG. 7), have re-established the connections betweenfilters, render engine 222 is ready to implement a user's instruction toexecute the media processing project.

Example Operation and Implementation(s)

The matrix switch described above is quite useful in that it allowsmultiple inputs to be directed to multiple outputs at any one time.These input can compete for a matrix switch output. The embodimentsdescribed below permit these competing inputs to be organized so thatthe inputs smoothly flow through the matrix switch to provide a desiredoutput. And, while the inventive programming techniques are described inconnection with the matrix switch as such is employed in the context ofmulti-media editing projects, it should be clearly understood thatapplication of the inventive programming techniques and structuresshould not be so limited only to application in the field of multi-mediaediting projects or, for that matter, multi-media applications or datastreams. Accordingly, the principles about to be discussed can beapplied to other fields of endeavor in which multiple inputs can becharacterized as competing for a particular output during a common timeperiod.

In the multi-media example below, the primary output of the matrixswitch is a data stream that defines an editing project that has beencreated by a user. Recall that this editing project can include multipledifferent sources that are combined in any number of different ways, andthe sources that make up a project can comprise audio sources, videosources, or both. The organization of the inputs and outputs of thematrix switch are made manageable, in the examples described below, by adata structure that permits the matrix switch to be programmed.

FIG. 9 shows an overview of a process that takes a user-defined editingproject and renders from it a data structure that can be used to programthe matrix switch.

Specifically, a user-defined editing project is shown generally at 900.Typically, when a user creates an editing project, they can select froma number of different multimedia clips that they can then assemble intoa unique presentation. Each individual clip represents a source ofdigital data or a source stream (e.g., multimedia content). Projects caninclude one or more sources 902. In defining their project, a user canoperate on sources in different ways. For example, video sources canhave transitions 904 and effects 906 applied on them. A transitionobject is a way to change between two or more sources. As discussedabove, a transition essentially receives as input, two or more streams,operates on them in some way, and produces a single output stream. Anexemplary transition can comprise, for example, fading from one sourceto another. An effect object can operate on a single source or on acomposite of sources. An effect essentially receives a single inputstream, operates on it in some way, and produces a single output stream.An exemplary effect can comprise a black-and-white effect in which avideo stream that is configured for presentation in color format isrendered into a video stream that is configured for presentation inblack and white format. Unlike conventional effect filters, effectobject 906 may well perform multiple effect tasks. That is, inaccordance with one implementation, an effect object (e.g., 906) mayactually perform multiple tasks on the received input stream, whereinsaid tasks would require multiple effect filters in a conventionalfilter graph system.

An exemplary user interface 908 is shown and represents what a usermight see when they produce a multimedia project with software executingon a computer. In this example, the user has selected three sources A,B, and C, and has assembled the sources into a project timeline. Theproject timeline defines when the individual sources are to be rendered,as well as when any transitions and/or effects are to occur.

In the discussion that follows, the notion of a track is introduced. Atrack can contain one or more sources or source clips. If a trackcontains more than one source clip, the source clips cannot overlap. Ifsource clips are to overlap (e.g. fading from one source to another, orhaving one source obscure another), then multiple tracks are used. Atrack can thus logically represent a layer on which sequential video isproduced. User interface 908 illustrates a project that utilizes threetracks, each of which contains a different source. In this particularproject source A will show for a period of time. At a defined time inthe presentation, source A is obscured by source B. At some later time,source B transitions to source C.

In accordance with the described embodiment, the user-defined editingproject 900 is translated into a data structure 910 that represents theproject. In the illustrated and described example, this data structure910 comprises a tree structure. It is to be understood, however, thatother data structures could be used. The use of tree structures torepresent editing projects is well-known and is not described here inany additional detail. Once the data structure 910 is defined, it isprocessed to provide a data structure 912 that is utilized to programthe matrix switch. In the illustrated and described embodiment, datastructure 912 comprises a grid from which the matrix switch can beprogrammed. It is to be understood and appreciated that other datastructures and techniques could, however, be used to program the matrixswitch without departing from the spirit and scope of the claimedsubject matter.

The processing that takes place to define data structures 910 and 912can take place using any suitable hardware, software, firmware, orcombination thereof. In the examples set forth below, the processingtakes place utilizing software in the form of a video editing softwarepackage that is executable on a general purpose computer.

Example Project

For purposes of explanation, consider FIG. 10 which shows project 908from FIG. 9 in a little additional detail. Here, a time line containingnumbers 0-16 provided adjacent the project to indicate when particularsources are to be seen and when transitions and effects (when present)are to occur. In the examples in this document, the following conventionexists with respect to projects, such as project 908. A priority existsfor video portions of the project such that as one proceeds from top tobottom, the priority increases. Thus, in the FIG. 10 example, source Ahas the lowest priority followed by source B and source C. Thus, ifthere is an overlap between higher and lower priority sources, thehigher priority source will prevail. For example, source B will obscuresource A from between t=4-8.

In this example, the following can be ascertained from the project 908and time line: from time t=0-4 source A should be routed to the matrixswitch's primary output; from t=4-12 source B should be routed to thematrix switch's primary output; from t=12-14 there should be atransition between source B and source C which should be routed to thematrix switch's primary output; and from t=14-16 source C should berouted to the matrix switch's primary output. Thus, relative to thematrix switch, each of the sources and the transition can becharacterized by where it is to be routed at any given time. Consider,for example, the table just below:

Object Routing for a given time C t = 0-12 (nowhere); t = 12-14(transition); t = 14-16 (primary output) B t = 0-4 (nowhere); t = 4-12(primary output); t = 12-14 (transition); t = 14-16 (nowhere) A t = 0-4(primary output); t = 4-16 (nowhere) Transition t = 0-12 (nowhere); t =12-14 (primary output); t = 14-16 (nowhere)

FIG. 11 shows an exemplary matrix switch 1100 that can be utilized inthe presentation of the user's project. Matrix switch 1100 comprisesmultiple inputs and multiple outputs. Recall that a characteristic ofthe matrix switch 1100 is that any of the inputs can be routed to any ofthe outputs at any given time. A transition element 1102 is provided andrepresents the transition that is to occur between sources B and C.Notice that the matrix switch includes four inputs numbered 0-3 andthree outputs numbered 0-2. Inputs 0-2 correspond respectively tosources A-C, while input 3 corresponds to the output of the transitionelement 1102. Output 0 corresponds to the switch's primary output, whileoutputs 1 and 2 are routed to the transition element 1102.

The information that is contained in the table above is the informationthat is utilized to program the matrix switch. The discussion presentedbelow describes but one implementation in which the informationcontained in the above table can be derived from the user's project timeline.

Recall that as a user edits or creates a project, software thatcomprises a part of their editing software builds a data structure thatrepresents the project. In the FIG. 9 overview, this was data structure910. In addition to building the data structure that represents theediting project, the software also builds and configures a matrix switchthat is to be used to define the output stream that embodies theproject. Building and configuring the matrix switch can include buildingthe appropriate graphs (e.g., a collection of software objects, orfilters) that are associated with each of the sources and associatingthose graphs with the correct inputs of the matrix switch. In addition,building and configuring the matrix switch can also include obtainingand incorporating additional appropriate filters with the matrix switch,e.g. filters for transitions, effects, and mixing (for audio streams).This will become more apparent below.

FIG. 12 shows a graphic representation of an exemplary data structure1200 that represents the project of FIG. 10. Here, the data structurecomprises a traditional hierarchical tree structure. Any suitable datastructure can, however, be utilized. The top node 1202 constitutes agroup node. A group encapsulates a type of media. For example, in thepresent example the media type comprises video. Another media type isaudio. The group node can have child nodes that are either tracks orcomposites. In the present example, three track nodes 1204, 1206, and1208 are shown. Recall that each track can have one or more sources. Ifa track comprises more than one source, the sources cannot overlap.Here, all of the sources (A, B, and C) overlap. Hence, three differenttracks are utilized for the sources. In terms of priority, the lowestpriority source is placed into the tree furthest from the left at 1204a. The other sources are similarly placed. Notice that source C (1208 a)has a transition 1210 associated with it. A transition object, in thisexample, defines a two-input/one output operation. When applied to atrack or a composition (discussed below in more detail), the transitionobject will operate between the track to which it has been applied, andany objects that are beneath it in priority and at the same level in thetree. A “tree level” has a common depth within the tree and belongs tothe same parent. Accordingly, in this example, the transition 1210 willoperate on a source to the left of the track on which source C resides,and beneath it in priority, i.e. source B. If the transition is appliedto any object that has nothing beneath it in the tree, it willtransition from blackness (and/or silence if audio is included).

Once a data structure representing the project has been built, in thiscase a hierarchical tree structure, a rendering engine processes thedata structure to provide another data structure that is utilized toprogram the matrix switch. In the FIG. 9 example, this additional datastructure is represented at 912. It will be appreciated and understoodthat the nodes of tree 1200 can include so-called meta information suchas a name, ID, and a time value that represents when that particularnode's object desires to be routed to the output, e.g. node 1204 a wouldinclude an identifier for the node associating it with source A, as wellas a time value that indicates that source A desires to be routed to theoutput from time t=0-8. This meta information is utilized to build thedata structure that is, in turn, utilized to program the matrix switch.

In the example about to be described below, a specific data structure inthe form of a grid is utilized. In addition, certain specifics aredescribed with respect to how the grid is processed so that the matrixswitch can be programmed. It is to be understood that the specificdescribed approach is for exemplary purposes only and is not intended tolimit application of the claims. Rather, the specific approachconstitutes but one way of implementing broader conceptual notionsembodied by the inventive subject matter.

FIGS. 13-18 represent a process through which the inventive grid isbuilt. In the grid about to be described, the x axis represents time,and the y axis represents layers in terms of priority that go fromlowest (at the top of the grid) to highest (at the bottom of the grid).Every row in the grid represents the video layer. Additionally, entriesmade within the grid represent output pins of the matrix switch. Thiswill become apparent below.

The way that the grid is built in this example is that the renderingengine does a traversal operation on the tree 1200. In this particularexample, the traversal operation is known as a “depth-first,left-to-right” traversal. This operation will layerize the nodes so thatthe leftmost track or source has the lowest priority and so on. Doingthe above-mentioned traversal on tree 1200 (FIG. 12), the first nodeencountered is node 1204 which is associated with source A. This is thelowest priority track or source. A first row is defined for the grid andis associated with source A. After the first grid row is defined, a gridentry is made and represents the time period for which source A desiresto be routed to the matrix switch's primary output.

FIG. 13 shows the state of a grid 1300 after this first processing step.Notice that from time t=0-8, a “0” has been placed in the grid. The “0”represents the output pin of the matrix switch—in this case the primaryoutput. Next, the traversal encounters node 1206 (FIG. 12) which isassociated with source B. A second row is thus defined for the grid andis associated with source B. After the second grid row is defined, agrid entry is made and represents the time period for which source Bdesires to be routed to the matrix switch's primary output.

FIG. 14 shows the state of grid 1300 after this second processing step.Notice that from time t=4-14, a “0” has been placed in the grid. Noticeat this point that something interesting has occurred which will beresolved below. Each of the layers has a common period of time (i.e.t=4-8) for which it desires to be routed to the matrix switch's primaryoutput. However, because of the nature of the matrix switch, only oneinput can be routed to the primary output at a time. Next, the traversalencounters node 1208 (FIG. 12) which is associated with source C. Inthis particular processing example, a rule is defined that sources ontracks are processed before transitions on the tracks are processedbecause transitions operate on two objects that are beneath them. Athird row is thus defined for the grid and is associated with source C.After the third row is defined, a grid entry is made and represents thetime period for which source C desires to be routed to the matrixswitch's primary output.

FIG. 15 shows the state of grid 1300 after this third processing step.Notice that from time t=12-16, a “0” has been placed in the grid. Next,the traversal encounters node 1210 (FIG. 12) which corresponds to thetransition. Thus, a fourth row is defined in the grid and is associatedwith the transition. After the fourth row is defined, a grid entry ismade and represents the time period for which the transition desires tobe routed to the matrix switch's primary output.

FIG. 16 shows the state of grid 1300 after this fourth processing step.Notice that from time t=12-14, a “0” has been placed in the grid for thetransition entry. The transition is a special grid entry. Recall thatthe transition is programmed to operate on two inputs and provide asingle output. Accordingly, starting at the transition entry in the gridand working backward, each of the entries corresponding to the same treelevel are examined to ascertain whether they contain entries thatindicate that they want to be routed to the output during the same timethat the transition is to be routed to the output. If grid entries arefound that conflict with the transition's grid entry, the conflictinggrid entry is changed to a value to corresponds to an output pin thatserves as an input to the transition element 1102 (FIG. 11). This isessentially a redirection operation. In the illustrated grid example,the transition first finds the level that corresponds to source C. Thislevel conflicts with the transition's grid entry for the time periodt=12-14. Thus, for this time period, the grid entry for level C ischanged to a switch output that corresponds to an input for thetransition element. In this example, a “2” is placed in the grid tosignify that for this given time period, this input is routed to outputpin 2. Similarly, continuing up the grid, the next level that conflictswith the transition's grid entry is the level that corresponds to sourceB. Thus, for the conflicting time period, the grid entry for level B ischanged to a switch output that corresponds to an input for thetransition element. In this example, a “1” is placed in the grid tosignify that for this given time period, this input is routed to outputpin 1 of the matrix switch.

FIG. 17 shows the state of the grid at this point in the processing.Next, a pruning function is implemented which removes any other lowerpriority entry that is contending for the output with a higher priorityentry. In the example, the portion of A from t=4-8 gets removed becausethe higher priority B wants the output for that time.

FIG. 18 shows the grid with a cross-hatched area that signifies thatportion of A's grid entry that has been removed.

At this point, the grid is in a state in which it can be used to programthe matrix switch. The left side entries—A, B, C, and TRANS representinput pin numbers 0, 1, 2, and 3 (as shown) respectively, on the matrixswitch shown in FIG. 11. The output pin numbers of the matrix switch aredesignated at 0, 1, and 2 both on the switch in FIG. 11 and within thegrid in FIG. 18. As one proceeds through the grid, starting with sourceA, the programming of the matrix switch can be ascertained as follows: Ais routed to output pin 0 of the matrix switch (the primary output) fromt=0-4. From t=4-16, A is not routed to any output pins. From t=0-4, B isnot routed to any of the output pins of the matrix switch. From t=4-12,B is routed to the primary output pin 0 of the matrix switch. Fromt=12-14, B is routed to output pin 1 of the matrix switch. Output pin 1of the matrix switch corresponds to one of the input pins for thetransition element 1102 (FIG. 11). From t=14-16, B is not routed to anyof the output pins of the matrix switch. From t=0-12, C is not routed toany of the output pins of the matrix switch. From t=12-14, C is routedto output pin 2 of the matrix switch. Output pin 2 of the matrix switchcorresponds to one of the input pins for the transition element 302(FIG. 3). From t=12-14 the transition element (input pin 3) is routed tooutput pin 0. From t=14-16, C is routed to output pin 0 of the matrixswitch.

As alluded to above, one of the innovative aspects of the matrix switch308 is its ability to seek to any point in a source, without having toprocess the intervening content serially through the filter. Rather,matrix switch 308 identifies an appropriate transition point and dumpsat least a subset of the intervening content, and continues processingfrom the seeked point in the content.

The ability of the matrix switch 308 to seek to any point in the mediacontent gives rise to certain performance enhancement heretoforeunavailable in computer implemented media processing systems. Forexample, generation of a filter graph by render engine 222 may take intoaccount certain performance characteristics of the media processingsystem which will execute the user-defined media processing project. Inaccordance with this example implementation, render engine 222 mayaccess and analyze the system registry of the operating system, forexample, to ascertain the performance characteristics of hardware and/orsoftware elements of the computing system implementing the mediaprocessing system, and adjust the filter graph construction to improvethe perceived performance of the media processing system by the user.Nonetheless, there will always be a chance that a particular instance ofa filter graph will not be able to process the media stream fast enoughto provide the desired output at the desired time, i.e., processing ofthe media stream bogs down leading to delays at the rendering filter. Insuch a case, matrix switch 308 will recognize that it is not receivingmedia content at the appropriate project time, and may skip certainsections of the project in an effort to “catch-up” and continue theremainder of the project in real time. According to one implementation,when matrix switch 308 detects such a lag in processing, it will analyzethe degree of the lag and issue a seek command to the source (throughthe source processing chain) to a future point in the project, whereprocessing continues without processing any further content prior to theseeked point.

Thus, for the editing project depicted in FIG. 10, the processingdescribed above first builds a data structure (i.e. data structure 1200in FIG. 12) that represents the project in hierarchical space, and thenuses this data structure to define or create another data structure thatcan be utilized to program the matrix switch.

FIG. 19 is a flow diagram that describes steps in a method in accordancewith the described embodiment. The method can be implemented in anysuitable hardware, software, firmware, or combination thereof. In theillustrated and described embodiment, the method is implemented insoftware.

Step 1900 provides a matrix switch. An exemplary matrix switch isdescribed above. Step 1902 defines a first data structure thatrepresents the editing project. Any suitable data structure can be used,as will be apparent to those of skill in the art. In the illustrated anddescribed embodiment, the data structure comprises a hierarchical treestructure having nodes that can represent tracks (having one or moresources), composites, transitions and effects. Step 1904 processes thefirst data structure to provide a second data structure that isconfigured to program the matrix switch. Any suitable data structure canbe utilized to implement the second data structure. In the illustratedand described embodiment, a grid structure is utilized. Exemplaryprocessing techniques for processing the first data structure to providethe second data structure are described above. Step 1906 then uses thesecond data structure to program the matrix switch.

Example Project with a Transition and an Effect

Consider project 2000 depicted in FIG. 20. In this project there arethree tracks, each of which contains a source, i.e. source A, B and C.This project includes an effect applied on source B and a transitionbetween sources B and C. The times are indicated as shown.

As the user creates their project, a data structure representing theproject is built. FIG. 21 shows an exemplary data structure in the formof a hierarchical tree 2100 that represents project 2000. There, thedata structure includes three tracks, each of which contains one of thesources. The sources are arranged in the tree structure in the order oftheir priority, starting with the lowest priority source on the left andproceeding to the right. There is an effect (i.e. “Fx”) that is attachedto or otherwise associated with source B. Additionally, there is atransition attached to or otherwise associated with source C.

In building the grid for project 2000, the following rule is employedfor effects. An effect, in this example, is a one-input/one-outputobject that is applied to one object—in this case source B. When theeffect is inserted into the grid, it looks for any one object beneath itin priority that has a desire to be routed to the primary output of thematrix switch at the same time. When it finds a suitable object, itredirects that object's output from the matrix switch's primary outputto an output associated with the effect.

As an example, consider FIG. 22 and the grid 2200. At this point in theprocessing of tree 2100, the rendering engine has incorporated entriesin the grid corresponding to sources A, B and the effect. It has done soby traversing the tree 2100 in the above-described way. In this example,the effect has already looked for an object beneath it in priority thatis competing for the primary output of the matrix switch. It found anentry for source B and then redirected B's grid entry to a matrix switchoutput pin that corresponds to the effect—here output pin 1.

As the render engine 222 completes its traversal of tree 2100, itcompletes the grid. FIG. 23 shows a completed grid 2200. Processing ofthe grid after that which is indicated in FIG. 22 takes placesubstantially as described above with respect to the first example.Summarizing, this processing though: after the effect is entered intothe grid and processed as described above, the traversal of tree 2100next encounters the node associated with source C. Thus, a row is addedin the grid for source C and an entry is made to indicate that source Cdesires the output from t=12-16. Next, the tree traversal encounters thenode associated with the transition. Accordingly, a row is added to thegrid for the transition and a grid entry is made to indicate that thetransition desires the output from t=12-14. Now, as described above, thegrid is examined to find two entries, lower in priority than thetransition and located at the same tree level as the transition, thatcompete for the primary output of the matrix switch. Here, those entriescorrespond to the grid entries for the effect and source C that occurfrom t=12-14. These grid entries are thus redirected to output pins ofthe matrix switch 308 that correspond to the transition—here pins 2 and3 as indicated. Next, the grid is pruned which, in this example, removesa portion of the grid entry corresponding to source A for t=4-8 becauseof a conflict with the higher-priority entry for source B.

FIG. 24 shows the resultant matrix switch that has been built andconfigured as the grid was being processed above. At this point, thegrid can be used to program the matrix switch. From the grid picture, itis very easy to see how the matrix switch 308 is going to be programmed.Source A will be routed to the matrix switch's primary output (pin 0)from t=0-4; source B will be redirected to output pin 1 (effect) fromt=4-14 and the effect on B will be routed to the output pin 0 fromt=4-12. From t=12-14, the effect and source C will be routed to outputpins corresponding to the transition (pins 2 and 3) and, accordingly,during this time the transition (input pin 4) will be routed to theprimary output (output pin 0) of the matrix switch. From t=14-16, sourceC will be routed to the primary output of the matrix switch.

It will be appreciated that as the software, in this case the renderengine 222, traverses the tree structure that represents a project, italso builds the appropriate graphs and adds the appropriate filters andgraphs to the matrix switch. Thus, for example, as the render engine 222encounters a tree node associated with source A, in addition to addingan entry to the appropriate grid, the software builds the appropriategraphs (i.e. collection of linked filters), and associates those filterswith an input of the matrix switch. Similarly, when the render engine222 encounters an effect node in the tree, the software obtains aneffect object or filter and associates it with the appropriate output ofthe matrix switch. Thus, in the above examples, traversal of the treestructure representing the project also enables the software toconstruct the appropriate graphs and obtain the appropriate objects andassociate those items with the appropriate inputs/outputs of the matrixswitch 308. Upon completion of the tree traversal and processing of thegrid, an appropriate matrix switch has been constructed, and theprogramming (i.e. timing) of inputs to outputs for the matrix switch hasbeen completed.

Treatment of “Blanks” in a Project

There may be instances in a project when a user leaves a blank in theproject time line. During this blank period, no video or audio isscheduled for play.

FIG. 25 shows a project that has such a blank incorporated therein. Ifthere is such a blank left in a project, the software is configured toobtain a “black” source and associate the source with the matrix switchat the appropriate input pin. The grid is then configured when it isbuilt to route the black source to the output at the appropriate timesand fade from the black (and silent) source to the next source at theappropriate times. The black source can also be used if there is atransition placed on a source for which there is no additional sourcefrom which to transition.

Audio Mixing

In the examples discussed above, sources comprising video streams werediscussed. In those examples, at any one time, only two video streamswere combined into one video stream. However, each project can, andusually does contain an audio component. Alternately, a project cancontain only an audio component. The audio component can typicallycomprise a number of different audio streams that are combined. Thediscussion below sets forth but one way of processing and combiningaudio streams.

In the illustrated example, there is no limit on the number of audiostreams that can be combined at any one time.

Suppose, for example, there is an audio project that comprises 5 tracks,A-E. FIG. 26 shows an exemplary project. The shaded portions of eachtrack represent the time during which the track is not playing. So, forexample, at t=0-4, tracks B, D, and E are mixed together and will play.From t=4-10, tracks A-E are mixed together and will play, and the like.

FIG. 27 shows the grid for this project at 2700. Since we are dealingwith this composition now, all of the effects and transitions includingthe audio mixing are only allowed to affect things in this composition.Thus, there is the concept of a boundary 2702 that prevents any actionsor operations in this composition from affecting any other grid entries.Note that there are other entries in the grid and that thepresently-illustrated entries represent only those portions of theproject that relate to the audio mixing function.

Grid 2700 is essentially set up in a manner similar to that describedabove with respect to the video projects. That is, for each track, a rowis added to the grid and a grid entry is made for the time period duringwhich the source on that track desires to be routed to the primaryoutput of the matrix switch. In the present example, grid entries aremade for sources A-E. Next, in the same way that a transition or effectwas allocated a row in the grid, a “mix” element is allocated a row inthe grid as shown and a grid entry is made to indicate that the mixelement desires to be routed to the primary output of the matrix switchfor a period of time during which two or more sources compete for thematrix switch's primary output. Note that in this embodiment, allocationof a grid row for the mix element can be implied. Specifically, whereasin the case of a video project, overlapping sources simply result inplaying the higher priority source (unless the user defines a transitionbetween them), in the audio realm, overlapping sources are treated as animplicit request to mix them. Thus, the mix element is allocated a gridrow any time there are two or more overlapping sources.

Once the mix element is allocated into the grid, the grid is processedto redirect any conflicting source entries to matrix switch output pinsthat correspond to the mix element. In the above case, redirection ofthe grid entries starts with pin 3 and proceeds through to pin 7. Thecorresponding matrix switch is shown in FIG. 28. Notice that all of thesources are now redirected through the mix element which is amulti-input/one output element. The mix element's output is fed backaround and becomes input pin 15 of the matrix switch. All of theprogramming of the matrix switch is now reflected in the grid 2700.Specifically, for the indicated time period in the grid, each of thesources is routed to the mix element which, in turn, mixes theappropriate audio streams and presents them to the primary output pin 0of the matrix switch.

Compositions

There are situations that can arise when building an editing projectwhere it would be desirable to apply an effect or a transition on just asubset of a particular project or track. Yet, there is no practicableway to incorporate the desired effect or transition. In the past,attempts to provide added flexibility for editing projects have beenmade in the form of so called “bounce tracks”, as will be appreciatedand understood by those of skill in the art. The use of bounce tracksessentially involves processing various video layers (i.e. tracks),writing or moving the processed layers or tracks to another location,and retrieving the processed layers when later needed for additionalprocessing with other layers or tracks. This type of processing can beslow and inefficient.

To provide added flexibility and efficiency for multi-media editingprojects, the notion of a composite or composition is introduced. Acomposite or composition can be considered as a representation of anediting project as a single track. Recall that editing projects can haveone or more tracks, and each track can be associated with one or moresources that can have effects applied on them or transitions betweenthem. In addition, compositions can be nested inside one another.

Example Project with Composite

Consider, for example, FIG. 29 which illustrates an exemplary project2900 having a composition 2902. In this example, composition 2902comprises sources B and C and a transition between B and C that occursbetween t=12-14. This composition is treated as an individual track orlayer. Project 2900 also includes a source A, and a transition betweensource A and composition 2902 at t=4-8. It will be appreciated thatcompositions can be much more complicated than the illustratedcomposition, which is provided for exemplary purposes only. Compositionsare useful because they allow the grouping of a particular set ofoperations on one or more tracks. The operation set is performed on thegrouping, and does not affect tracks that are not within the grouping.To draw an analogy, a composition is similar in principle to amathematical parenthesis. Those operations that appear within theparenthesis are carried out in conjunction with those operations thatare intended to operate of the subject matter of the parenthesis. Theoperations within the parenthesis do not affect tracks that do notappear within the parenthesis.

In accordance with the processing that is described above in connectionwith FIG. 19, a first data structure is defined that represents theediting project. FIG. 30 shows an exemplary data structure 3000 in theform of a hierarchical tree structure. In this example, group node 3002includes two children—track node 3004 and composite node 3006. Tracknode 3004 is associated with source A. Composite node 3006 includes twochildren—track nodes 3008 and 3010 that are respectively associated withsources B (3008 a) and C (3010 a). A transition T2 (3012) is applied onsource C and a transition T1 (3014) is applied on composition 3006.

Next, data structure 3000 is processed to provide a second datastructure that is configured to program the matrix switch. Note that asthe data structure is being programmed, a matrix switch is being builtand configured at the same time. In this example, the second datastructure comprises a grid structure that is assembled in much the sameway as was described above. There are, however, some differences and,for purposes of understanding, the complete evolution of the gridstructure is described here. In the discussion that follows, thecompleted matrix switch is shown in FIG. 38.

When the rendering engine initiates the depth-first, left-to-righttraversal of data structure 3000, the first node it encounters is tracknode 3004 which is associated with source A. Thus, a first row of thegrid is defined and a grid entry is made that represents the time periodfor which source A desires to be routed to the matrix switch's primaryoutput pin.

FIG. 31 shows the state of a grid 3100 after this first processing step.Next the traversal of data structure 3000 encounters the composite node3006. The composite node is associated with two tracks-track 3008 andtrack 3010. Track 3008 is associated with source B. Accordingly, asecond row of the grid is defined and a grid entry is made thatrepresents the time period for which source B desires to be routed tothe matrix switch's primary output pin. Additionally, since B is amember of a composition, meta-information is contained in the grid thatindicates that this grid row defines one boundary of the composition.This meta-information is graphically depicted with a bracket thatappears to the left of the grid row.

FIG. 32 shows the state of grid 3100 after this processing step. Next,the traversal of data structure 3000 encounters node 3010 which isassociated with source C. Thus, a third row of the grid is added and agrid entry is made that represents the time period for which source Cdesires to be routed to the matrix switch's primary output pin.

FIG. 33 shows the state of grid 3100 after this processing step. Noticethat the bracket designating the composition now encompasses the gridrow associated with source C. The traversal next encounters node 3012which is the node associated with the second transition T2. Thus, as inthe above example, a grid row is added for the transition and a gridentry is made that represents the time period for which the transitiondesires to be routed to the matrix switch's primary output pin.

FIG. 34 shows the state of grid 3100 after this processing step. Noticethat the bracket designating the composition is now completed andencompasses grid row entries that correspond to sources B and C and thetransition between them. Recall from the examples above that atransition, in this example, is programmed to operate on two inputs andprovide a single output. In this instance, and because the transitionoccurs within a composition, the transition is constrained by a rulethat does not allow it to operate on any elements outside of thecomposition. Thus, starting at the transition entry and working backwardthrough the grid, entries at the same tree level and within thecomposition (as designated by the bracket) are examined to ascertainwhether they contain entries that indicate that they want to be routedto the output during the same time that the transition is to be routedto the output. Here, both of the entries for sources B and C haveportions that conflict with the transition's entry. Accordingly, thoseportions of the grid entries for sources B and C are redirected orchanged to correspond to output pins that are associated with atransition element that corresponds to transition T2.

FIG. 35 shows the state of grid 3100 after this processing step. Thetraversal next encounters node 3014 which is the node that is associatedwith the transition that occurs between source A and composition 2902(FIG. 29). Processing of this transition is similar to processing of thetransition immediately above except for the fact that the transitiondoes not occur within the composition. Because the transition occursbetween the composition and another source, one of the inputs for thetransition will be the composition, and one of the inputs will be sourceA (which is outside of the composition). Thus, a grid row is added forthis transition and a grid entry is made that represents the time periodfor which the transition desires to be routed to the matrix switch'sprimary output pin.

FIG. 36 shows the state of grid 3100 after this processing step. At thispoint then, the grid is examined for entries that conflict with theentry for transition T1. One conflicting grid entry is found for the rowthat corresponds to source B (inside the composition) and one thatcorresponds to source A (outside the composition). Accordingly, thoseportions of the grid row that conflict with transition T1 are changed orredirected to have values that are associated with output pins of thematrix switch that are themselves associated with a transition elementT1. In this example, redirection causes an entry of “3” and “4” to beinserted as shown.

FIG. 37 shows the state of grid 3100 after this processing step. Ifnecessary, a pruning operation would further ensure that the grid has nocompeting entries for the primary output of the matrix switch. Theassociated input pin numbers of the matrix switch are shown to the leftof grid 3100.

FIG. 38 shows a suitably configured matrix switch that has been build inaccordance with the processing described above. Recall that, as datastructure 3000 (FIG. 30) is processed by the rendering engine, a matrixswitch is built and configured in parallel with the building andprocessing of the grid structure that is utilized to program the matrixswitch. From the matrix switch and grid 3100 of FIG. 37, the programmingof the switch can be easily ascertained.

FIG. 38 a shows an exemplary data structure that represents a projectthat illustrates the usefulness of composites. In this example, theproject can mathematically be represented as follows:(Fx−noisy(A Tx−Blend B))Tx−Blend C

Here, an effect (noisy) is applied to A blended with B, the result ofwhich is applied to a blend with C. The composite in this example allowsthe grouping of the things beneath it so that the effect (noisy), whenit is applied, is applied to everything that is beneath it. Notice thatwithout the composite node, there is no node where an effect can beapplied that will affect (A Tx−Blend B). Hence, in this example,operations that appear within the parenthesis are carried out on tracksthat appear within the parenthesis. Those operations do not affecttracks that are not within the parenthesis.

FIG. 39 is a flow diagram that described steps in a method in accordancewith one embodiment. The method can be implemented in any suitablehardware, software, firmware, or combination thereof. In thepresently-described example, the method is implemented in software.

Step 3900 defines a multimedia editing project that includes at leastone composite. The composite represents multiple tracks as a singletrack for purposes of the processing described just below. It isimportant to note that, in the processing described just below, andbecause of the use of composites, the extra processing that is requiredby bounce tracks is avoided (i.e. operating on two tracks, moving theoperation result to another location, and retrieving the operationresult when later needed). This reduces the processing time that isrequired to render a multi-media project. Step 3902 defines a first datastructure that represents the editing project. Any suitable datastructure can be utilized. In the present example, a data structure inthe form of a hierarchical tree is utilized. An exemplary tree is shownin FIG. 30. Step 3904 processes the first data structure to provide asecond data structure that is configured to program a matrix switch. Inthe illustrated example, the second data structure comprises a gridstructure. Exemplary processing is described in the context of FIGS.30-37. Step 3906 then programs the matrix switch using the second datastructure.

CONCLUSION

The described embodiments can be used to provide improvements overprevious multi-media editing systems. Various efficiencies are achievedthat reduce the processing times and can thereby improve the userexperience when using multi-media project editing software applications.

Although the invention has been described in language specific tostructural features and/or methodological steps, it is to be understoodthat the invention defined in the appended claims is not necessarilylimited to the specific features or steps described. Rather, thespecific features and steps are disclosed as preferred forms ofimplementing the claimed invention.

1. A multi-media processing system comprising: means for representingmultiple tracks as a single track, each of the multiple tracks beingcapable of being associated with one or more digital data streams, thesingle track comprising a portion of a multi-media editing project;means for defining a first data structure that represents themulti-media editing project; means for providing a software-implementedmatrix switch that is programmable to route multiple switch inputs tomultiple switch outputs, at least two of the inputs being capable ofcompeting for a single output during a common time period so that no twoinputs are routed to said single output during the common time period,the single output being configured to provide a data stream defined bythe multi-media editing project; means for processing the first datastructure to provide a second data structure that contains data that canbe used to program the matrix switch so that multiple switch inputs arerouted to multiple switch outputs; wherein said means for processing thefirst data structure to provide the second data structure comprisesmeans for processing the first data structure to provide a gridstructure and means for changing at least one data value within the gridstructure so that no two inputs are routed to said single output duringa common time period.
 2. The multi-media editing system of claim 1,wherein said means for processing the first data structure comprisesmeans for configuring the matrix switch so that it receives at least onesource stream at one of its inputs.
 3. The multi-media editing system ofclaim 2, wherein said means for configuring comprises means for buildinga graph comprising a collection of filters configured to process digitaldata to provide a source stream.
 4. The multi-media editing system ofclaim 2, wherein said means for processing the first data structurecomprises means for providing at least one element configured to receiveat least one output stream from the matrix switch, the one elementhaving an output that is routed back to an input of the switch.
 5. Themulti-media editing system of claim 4, wherein said element comprises atransition element that is configured to change between at least onesource stream and at least one other source stream.
 6. The multi-mediaediting system of claim 4, wherein said element comprises an effectelement that is configured to operate on a source stream to produce adifferent source stream.
 7. The multi-media editing system of claim 1,wherein said means for defining the first data structure comprises meansfor defining a hierarchical tree structure.
 8. The multi-media editingsystem of claim 7, wherein the hierarchical tree structure comprisesnodes that represent tracks, each track being associated with one ormore data stream sources.
 9. The multi-media editing system of claim 7,wherein the hierarchical tree structure comprises nodes at least one ofwhich represents a composite which embodies said multiple tracksrepresented as a single track.
 10. The multi-media editing system ofclaim 9, wherein the hierarchical tree structure can comprise nodes thatrepresent transitions between sources within a composite.
 11. Themulti-media editing system of claim 9, wherein the hierarchical treestructure can comprise nodes that represent effects applied on sourceswithin a composite.
 12. The multi-media editing system of claim 9,wherein the hierarchical tree structure can comprise nodes thatrepresent transitions between the composite and a source.
 13. Themulti-media editing system of claim 9, wherein the hierarchical treestructure can comprise nodes that represent effects applied on thecomposite.
 14. The multi-media editing system of claim 8, wherein thedata stream sources with which one or more of the tracks are associatedcomprise video data stream sources.
 15. The multi-media editing systemof claim 1, wherein said means for processing the first data structureto provide the second data structure comprises means for processing thefirst data structure to provide a grid structure, the grid structurecontaining multiple rows, individual rows representing input pins of thematrix switch and being associated with individual data stream sources,each row containing at least one value that is associated with an outputpin associated with said single output of the matrix switch.
 16. Asystem comprising: means for defining a multimedia editing projectresponsive to input received from a user, the project being capable ofcontaining at least one composite comprising multiple tracks; means forrepresenting the editing project as a hierarchical tree structurecomprising multiple nodes, at least one of the nodes representing saidat least one composite; means for processing the hierarchical treestructure to provide a grid structure containing data that defines anassociation between inputs, outputs and a time line defined by theediting project; means for providing a matrix switch having multipleinputs and multiple outputs that correspond to the respective inputs andoutputs of the grid structure; and means for using the grid structure toprogram routing of the matrix switch's inputs to the matrix switch'soutputs for the given time line.
 17. The system of claim 16, wherein theproject can contain at least one composite nested inside anothercomposite.
 18. The system of claim 16, wherein the project can contain:multiple tracks; transitions between tracks; transitions between a trackand a composite and vice versa; and effects applied on tracks andcomposites.
 19. The system of claim 16, further comprising means forchanging at least one data value within the grid structure to ensurethat only one stream is routed to a primary input of the matrix switchat a time.
 20. A multi-media processing system comprising: means forrepresenting multiple tracks as a single track, each of the multipletracks being capable of being associated with one or more digital datastreams, the single track comprising a portion of a multi-media editingproject; means for defining a first data structure that represents themulti-media editing project; means for providing a software-implementedmatrix switch that is programmable to route multiple switch inputs tomultiple switch outputs, at least two of the inputs being capable ofcompeting for a single output during a common time period, the singleoutput being configured to provide a data stream defined by themulti-media editing project; means for processing the first datastructure to provide a second data structure that contains data that canbe used to program the matrix switch so that multiple switch inputs arerouted to multiple switch outputs; and means for operating on the seconddata structure so that no two inputs are routed to said single outputduring a common time period; wherein said means for processing the firstdata structure to provide the second data structure comprises means forprocessing the first data structure to provide a grid structure, thegrid structure containing multiple rows, individual rows representinginput pins of the matrix switch and being associated with individualdata stream sources, each row containing at least one value that isassociated with an output pin associated with said single output of thematrix switch; and wherein said means for operating on the second datastructure comprises means for changing at least one value in at leastone of the rows of the grid structure.